The present invention relates to electronic camera apparatus and methods based on signal processing techniques for (i) determining the distance of objects from a camera system, (ii) rapid autofocusing of camera systems, and (iii) obtaining improved focus images from blurred images. In general, this invention relates to system parameter estimation and signal restoration in any linear shift-invariant system, i.e., a system that performs convolution operation on the input signal.
A wide variety of apparatus and methods are known for finding range (i.e. distance) of objects. A survey of these apparatus and methods is found in an article by R. A. Jarvis titled "A perspective on range finding techniques for computer vision" in the March 1983 issue of IEEE Transactions on Pattern analysis and Machine Intelligence, PAMI-5, No. 2, pages 122-139.
A common technique of finding the distance of an object from a camera involves focusing the object's image on the image detector. The distance is then determined from the camera setting. This technique is called depth-from-focusing. A comparative study of different depth-from-focusing methods is reported by E. Krotkov in an article titled "Focusing" in the October 1987 issue of International Journal of Computer Vision, Volume 1, No. 3, pages 223-238.
The depth-from-focusing technique involves (i) acquiring a large number (about 20 or more) of images for different camera settings, (ii) computing a focus measure for each of these images, and, (iii) determining the image from which the focus measure is a maximum and determining distance from the corresponding camera setting. The major drawbacks of this technique are (i) it is very slow because it involves acquiring many images for different camera settings, and (ii) it requires large computational resources in terms of memory space and processing power.
Two new methods for finding distance of objects are described by A. P. Pentland in a paper titled "A new Sense for Depth of Field" published in the Proceedings of the International Joint Conference on Artificial Intelligence, August, 1985. The same paper was revised and republished with minor changes in July 1987 in IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. PAMI-9, No. 4, pages 523-531.
The first method of Pentland uses a single image of an object. The object must contain a bright surface patch adjacent to a dark surface patch with a common straight border in between. Such an object produces a straight edge which is a step discontinuity in the focused image. A closely related method is disclosed by P. Grossman in a paper titled "Depth from Focus" published in Pattern Recognition Letters, Vol. 5, pages 63-69, Jan. 1987. A much improved and more general version of these same methods is disclosed by this inventor in a paper titled "Depth Recovery from Blurred Edges," published in the Proceedings of IEEE Computer Society conference on Computer Vision and Pattern Recognition, June 1988, Pages 498-503. However, objects having bright and dark surface patches with a common straight border in between are rare in this world. Therefore it is not applicable in most cases.
The second method of Pentland is based on comparing two images, one image formed with a very small (pin-hole) aperture and the other image formed with a normal aperture. Application of this method poses some serious practical difficulties. A very small aperture increases the effects of diffraction which distorts the image. Further, a smaller aperture gathers lesser light and therefore increases exposure period. These problems make the method inaccurate, slow, and of limited use in practical applications.
In addition to determining the distance of objects, the present invention advances technology for rapid autofocusing of electronic cameras. It has been known to autofocus cameras by several methods. One method for autofocusing has been disclosed in Japanese Patent No. 62284314 dated 1987. This method is believed to be incorrect in theory, inaccurate, and, consequently, of limited use in practice because of the following reasons. The method disclosed in the '314 Japanese Patent involves moving the lens with respect to a fixed object. Therefore, the correspondence between the two images of a point object, e.g., the first image taken before moving the lens and the second image taken after moving the lens, cannot be established. This problem, which is referred to as correspondence problem, is well known in stereo-vision. Furthermore, the method as set forth in the '314 Japanese disclosure does not provide a magnification correction step so that errors are introduced into the process. Moreover, the method referred to above is very restricted in scope since it involves changing only one camera parameter--the position of the lens.
Further drawbacks with regard to the method set forth in Japanese '314 disclosure include the use of at least three pictures, and reliance on only high spatial frequencies--and even then only a fixed spatial frequency for autofocusing.
Another autofocusing method titled "Focal Point Detector" has been disclosed by Takeshi Baba, et al. in Japanese Patent No. 63-127217 laid open on May 31, 1988. The Japanese '217 disclosure suffers from the same drawbacks a set forth above with regard to the Japanese '314 disclosure, and is applicable only when the Modulation Transfer Function (MTF) is a Gaussian function. The Gaussian is a very crude approximation to the actual MTF of the camera system. Therefore, the method of '217 Japanese disclosure is also subject to error.
A previous method and apparatus for finding the distance of objects and autofocusing has been disclosed by the same inventor in commonly-owned copending U.S. patent application Ser. No. 126,407 filed on Nov. 27, 1987. The previous method involves processing two images of an object, the images being acquired with different camera settings. The camera setting differs in any of the three camera parameters: (i) position of the image detector, (ii) focal length of the camera system, and (iii) diameter of the camera's aperture. Two constraints, which are mathematical expressions (i.e. binding relations), were then formulated. The first constraint was expressed solely in terms of the two observed images, while the second constraint was expressed solely in terms of the known camera parameter values and camera characteristics. These constraints or equations involve two unknowns, one unknown for each of the two images. Equations had then been solved simultaneously, and the distance subsequently determined from the solution of the unknown. However, the unknown corresponding to an image was actually an intermediate parameter, .sigma., which characterized the point spread function of the camera system. This intermediate parameter was related to the spread or 37 blur" of the focused image and was based on an assumption that the point spread function has a fixed or predetermined shape or form which can be represented by a Gaussian expression.
This earlier method characterized the Point Spread Function (PSF) by a single intermediate parameter, .sigma., which is the standard deviation of the PSF. This is an inadequate characterization in the presence of diffraction and lens aberrations. The previous system set forth by the inventor in his copending application requires that means be provided in the camera system for measuring the camera parameters. The previous system does not consider the spectral content of the light as a parameter. Moreover, the previous system does not provide for performing uncertainty analysis or error propagation analysis during processing, nor was it applicable to every linear shift-invariant system for estimating a number of system parameters.
It is, therefore, an object of the present invention to provide a depth-from-focusing approach which involves the acquisition and processing of only two images.
It is also an object of the present invention to eliminate the correspondence problem associated with stereo-vision techniques.
It is yet a further object of the present invention to provide an approach which is applicable to all linear shift-variant systems.
Yet another object of the present invention is to provide a simpler and less expensive apparatus for the processing of two images to determine both object distance and camera parameters.
Another object of the present invention is to eliminate the need for an intermediate parameter to characterize the Point Spread Function of the camera system. Another object of the present invention is to eliminate the need for quantitatively measuring the parameters in the camera apparatus.
Another object of the present invention is to provide the means for a single calculation or process step to determine the distance of objects, the optical transfer function, and/or camera parameter values.
An additional object of the present invention is to provide a means for considering the spectral content of the light as it relates to the camera parameters.
Another object of the present invention is to provide rapid autofocusing of electronic cameras, and obtaining improved-focus images from blurred images.
Another object of the present invention is a method for estimating system parameters and recovering input signal in any instrument that can be modeled as a linear shift-invariant system, i.e., a system that performs convolution operation.
Other objects will be known from the present disclosure and it is not intended to be in any way limited by setting forth some of the objects of the present invention.